Intelligent vehicle point of focus communication

ABSTRACT

Herein is disclosed a virtual embodiment display system comprising one or more image sensors, configured to receive one or more images of a vehicle occupant; one or more processors, configured to determine a gaze direction of the vehicle occupant from the one or more images; select a display location corresponding to the determined gaze direction; and control an image display device to display a virtual embodiment of an intelligent agent at the display location; the image display device, configured to display the virtual embodiment of the intelligent agent at the selected display location according to the one or more processors.

TECHNICAL FIELD

Various aspects of the disclosure relate generally to the use of anintelligent agent for interaction between artificial intelligence andhumans.

BACKGROUND

As vehicles assume functions that had previously been reserved fordrivers, thereby becoming autonomous or semi-autonomous vehicles,communication between vehicle occupant and vehicle assumes greaterimportance. Existing models for such communication tends to reflect adesire for the vehicle to provide a variety of information to the driveras well as the passengers, such as in the context of one or moredisplays, which may display data related to the vehicle's autonomous orsemi-autonomous actions. Occupants may similarly provide information tothe vehicle through one or more controls (buttons and the like) or byinputting information on a screen. In addition, speech recognitionabilities have permitted at least some level of voice-basedcommunication between humans and machines. Whether screen-based orspeech-based, natural and fluent communication requires a point of focusfor communication between the occupant and the vehicle, and the lack ofa point of focus is detrimental to communication.

SUMMARY

Herein is disclosed a virtual embodiment display system including one ormore image sensors, configured to receive one or more images of avehicle occupant; one or more processors, configured to determine a gazedirection of the vehicle occupant from the one or more images; select adisplay location corresponding to the determined gaze direction; andcontrol an image display device to display a tag at the displaylocation; the image display device, configured to display the tag at theselected display location according to the one or more processors.

BRIEF DESCRIPTION OF THE DRAWINGS

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures. The drawings are not necessarily to scale, emphasis insteadgenerally being placed upon illustrating aspects of the disclosure. Inthe following description, some aspects of the disclosure are describedwith reference to the following drawings, in which:

FIG. 1 depicts a conventional occupant-vehicle interface;

FIG. 2 depicts an occupant-vehicle interface according to a first aspectof the disclosure;

FIG. 3 depicts an occupant-vehicle interface according to a secondaspect of the disclosure;

FIG. 4 depicts an occupant-vehicle interface according to a third aspectof the disclosure;

FIG. 5 depicts a vehicle visual tag system according to a fourth aspectof the disclosure;

FIG. 6 depicts a physical embodied agent's correlation of an occupantgaze with an occupant gesture;

FIG. 7 depicts an additional configuration of the first aspect of thedisclosure;

FIG. 8 depicts an additional configuration of the second aspect of thedisclosure;

FIG. 9 depicts an additional configuration of the third aspect of thedisclosure;

FIG. 10 depicts a processing workflow;

FIG. 11 shows elements of a gaze calculation;

FIG. 12 depicts an operation of the Fusion & Adaptation Module (FAM)block according to an aspect of the disclosure;

FIG. 13 depicts the Passenger Interaction Engine (PIE) according to anaspect of the disclosure;

FIG. 14 depicts a procedure for identifying a desired location forprojection or display of the visual tag;

FIG. 15 depicts a determination of a preferred display location based onthe passenger gaze direction with respect to the amount of deviationfrom line of sight to each available display in the vehicle;

FIG. 16 depicts the Agent Embodiment Controller (“AEC”);

FIG. 17 depicts a vehicle tag system;

FIG. 18 depicts a vehicle embodied agent system; and

FIG. 19 depicts a method of displaying a tag.

DESCRIPTION

The following detailed description refers to the accompanying drawingsthat show, by way of illustration, specific details and aspects in whichthe disclosure may be practiced. These aspects are described insufficient detail to enable those skilled in the art to practice thedisclosure. Other aspects may be utilized and structural, logical, andelectrical changes may be made without departing from the scope of thedisclosure. The various aspects are not necessarily mutually exclusive,as some aspects can be combined with one or more other aspects to formnew aspects. Various aspects are described in connection with methodsand various aspects are described in connection with devices. However,it may be understood that aspects described in connection with methodsmay similarly apply to the devices, and vice versa.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any aspect of the disclosure describedherein as “exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects of the disclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

The terms “at least one” and “one or more” may be understood to includea numerical quantity greater than or equal to one (e.g., one, two,three, four, [ . . . ], etc.). The term “a plurality” may be understoodto include a numerical quantity greater than or equal to two (e.g., two,three, four, five, [ . . . ], etc.).

The phrase “at least one of” with regard to a group of elements may beused herein to mean at least one element from the group consisting ofthe elements. For example, the phrase “at least one of” with regard to agroup of elements may be used herein to mean a selection of: one of thelisted elements, a plurality of one of the listed elements, a pluralityof individual listed elements, or a plurality of a multiple of listedelements.

The words “plural” and “multiple” in the description and the claimsexpressly refer to a quantity greater than one. Accordingly, any phrasesexplicitly invoking the aforementioned words (e.g. “a plurality of[objects]”, “multiple [objects]”) referring to a quantity of objectsexpressly refers more than one of the said objects. The terms “group(of)”, “set [of]”, “collection (of)”, “series (of)”, “sequence (of)”,“grouping (of)”, etc., and the like in the description and in theclaims, if any, refer to a quantity equal to or greater than one, i.e.one or more. The terms “proper subset”, “reduced subset”, and “lessersubset” refer to a subset of a set that is not equal to the set, i.e. asubset of a set that contains less elements than the set.

The term “data” as used herein may be understood to include informationin any suitable analog or digital form, e.g., provided as a file, aportion of a file, a set of files, a signal or stream, a portion of asignal or stream, a set of signals or streams, and the like. Further,the term “data” may also be used to mean a reference to information,e.g., in form of a pointer. The term data, however, is not limited tothe aforementioned examples and may take various forms and represent anyinformation as understood in the art.

The term “processor” or “controller” as, for example, used herein may beunderstood as any kind of entity that allows handling data, signals,etc. The data, signals, etc. may be handled according to one or morespecific functions executed by the processor or controller.

A processor or a controller may thus be or include an analog circuit,digital circuit, mixed-signal circuit, logic circuit, processor,microprocessor, Central Processing Unit (CPU), Neuromorphic ComputerUnit (NCU), Graphics Processing Unit (GPU), Digital Signal Processor(DSP), Field Programmable Gate Array (FPGA), integrated circuit,Application Specific Integrated Circuit (ASIC), etc., or any combinationthereof. Any other kind of implementation of the respective functions,which will be described below in further detail, may also be understoodas a processor, controller, or logic circuit. It is understood that anytwo (or more) of the processors, controllers, or logic circuits detailedherein may be realized as a single entity with equivalent functionalityor the like, and conversely that any single processor, controller, orlogic circuit detailed herein may be realized as two (or more) separateentities with equivalent functionality or the like.

The term “system” (e.g., a drive system, a position detection system,etc.) detailed herein may be understood as a set of interactingelements, the elements may be, by way of example and not of limitation,one or more mechanical components, one or more electrical components,one or more instructions (e.g., encoded in storage media), one or morecontrollers, etc.

A “circuit” as user herein is understood as any kind oflogic-implementing entity, which may include special-purpose hardware ora processor executing software. A circuit may thus be an analog circuit,digital circuit, mixed-signal circuit, logic circuit, processor,microprocessor, Central Processing Unit (“CPU”), Graphics ProcessingUnit (“GPU”), Neuromorphic Computer Unit (NCU), Digital Signal Processor(“DSP”), Field Programmable Gate Array (“FPGA”), integrated circuit,Application Specific Integrated Circuit (“ASIC”), etc., or anycombination thereof. Any other kind of implementation of the respectivefunctions which will be described below in further detail may also beunderstood as a “circuit.” It is understood that any two (or more) ofthe circuits detailed herein may be realized as a single circuit withsubstantially equivalent functionality, and conversely that any singlecircuit detailed herein may be realized as two (or more) separatecircuits with substantially equivalent functionality. Additionally,references to a “circuit” may refer to two or more circuits thatcollectively form a single circuit.

As used herein, “memory” may be understood as a non-transitorycomputer-readable medium in which data or information can be stored forretrieval. References to “memory” included herein may thus be understoodas referring to volatile or non-volatile memory, including random accessmemory (“RAM”), read-only memory (“ROM”), flash memory, solid-statestorage, magnetic tape, hard disk drive, optical drive, etc., or anycombination thereof. Furthermore, it is appreciated that registers,shift registers, processor registers, data buffers, etc., are alsoembraced herein by the term memory. It is appreciated that a singlecomponent referred to as “memory” or “a memory” may be composed of morethan one different type of memory, and thus may refer to a collectivecomponent including one or more types of memory. It is readilyunderstood that any single memory component may be separated intomultiple collectively equivalent memory components, and vice versa.Furthermore, while memory may be depicted as separate from one or moreother components (such as in the drawings), it is understood that memorymay be integrated within another component, such as on a commonintegrated chip.

The term “tag” as used herein to indicate an identifier of theartificial intelligence. The tag may take any form whatsoever. Accordingto one aspect of the disclosure, the tag may have a human or near-humanform. According to another aspect of the disclosure, the tag may have arepresentative form, such as, but not limited to, a cartoon, a drawing,a comic, an avatar, or other figure. According to another aspect of thedisclosure, the tag may have an abstract form, such as one or moregeometric shapes, a design, a logo, or otherwise. The tag may be avirtual embodiment of the intelligent agent.

Current research efforts and industry trends focus on positioning theautonomous vehicle as a trusted social companion, suggesting an effortto help humans perceive intelligent vehicles as representations of asentient being. This requires establishing a natural means ofcommunication between humans and autonomous vehicle. Without firstestablishing a natural means of communication between humans andautonomous vehicles, the perception of the vehicle as a representationof a being is unlikely to be achieved. One meaningful component of thiscommunication is a dialogue management system within the autonomousvehicle. An additional meaningful component is a representation withinthe autonomous vehicle of the vehicle intelligence, thus providing acentral point of focus for one or more human occupants to communicateand interact with the autonomous vehicle. It may be further necessary tocreate the representation in a configurable way, such that adaptationcan be performed as research develops, and to apply to a variety ofvehicle configurations and autonomous vehicle interior designs.

As autonomous vehicles are developed, it has been known to create apoint of interaction between a vehicle occupant and the autonomousvehicle in the form of a screen, said screen typically providingunidirectional information to the occupant regarding the vehicle'sintentions or actions. Various autonomous vehicle implementations havearranged said information in a single screen, or possibly throughoutmultiple screens within the vehicle. Said screen or screens may provideinformation such as, for example, vehicle speed, vehicle surroundings,and any vehicle maneuvers performed by the autonomous vehicle. Suchscreen interfaces have been used in performance and/or tests of avariety of autonomous vehicle levels, including Level 3, Level 4, andLevel 5 driving. Although providing information in this manner to theoccupant may have some utility, it does not approximate human to humaninteraction. As such, effective communication may require severaladditional translational steps, or information may be lost altogether.

Currently used autonomous vehicles may lack a point of focus within theautonomous vehicle for interaction between one or more occupants and theintelligent agent of the autonomous vehicle. Conventionally usedcommunication interfaces, such as screens delivering data, may not besuitable for an occupant to make natural eye contact while communicatingwith the autonomous vehicle, if they lack an embodiment of theintelligent agent of the autonomous vehicle. Furthermore, unless apassenger believes that the conventionally used communication device hasthe ability to see and understand paralinguistic features of language,the passenger will be less likely to use these features which, if usedby the passenger, could facilitate communication between the passengerand the vehicle intelligence. Aspects of this disclosure include virtualand/or physical embodiments of the intelligent agent that serve as apoint of focus system for occupants within the autonomous vehicle, aswell as system components that are designed to capture and utilizeoccupant movements and/or interactions with these embodiments in variousconfigurations. These may result in a more natural manner of interactingwith the intelligent agent of the autonomous vehicle and renderautonomous vehicle technology closer to becoming a trusted companion forhuman beings.

As autonomous vehicle technology is improved, it is anticipated thatautonomous vehicles will perform with Level 4 and Level 5 capability,for which improved communication between occupant and autonomous vehiclemay be beneficial. In addition, in-cabin experiences for vehicleoccupants may become key differentiators for fleet services that will beoffered by various vehicles. Increased occupant satisfaction, as well asincreased or improved occupant-to-autonomous vehicle communication maybe reached by improving communication interfaces to approximate anatural human-to-human communication. Furthermore, by providingoccupants with a point of focus, the gaze of the occupants may beconstrained in order to improve sensing accuracy of in-cabin humanunderstanding scenarios. Such scenarios may be occasions of humanmovement or human gaze behavior, for which the autonomous vehicle mayperform a behavioral analysis in order to gain an understanding of thehuman occupant's instructions, impressions or desires. Such behavioralanalysis may be more successful where the occupant's gaze is generatedwith a particular perspective in mind, rather than when the occupant'sgaze has no specific point of focus. The principles described herein maypermit a vehicle occupant to achieve a higher level of integration with,and control of, the autonomous vehicle system.

FIG. 1 depicts a known vehicle information system 100, as viewed fromwithin an interior of the vehicle. A vehicle with some level ofautonomous vehicle capability may include a communication interface 102,on which information is provided to a vehicle occupant. Thecommunication interface 102 may be subdivided into a number ofinterfaces with one or more dedicated functions, such as depicted in104, 106, and 108. Under this configuration, and occupant within thevehicle turns to the communication interface 102 to obtain informationabout the vehicle and/or any autonomous vehicle functions orcapabilities. Information provided to the vehicle may be provided viathe communication interface 102.

FIG. 2 depicts a vehicle visual tag system according to a first aspectof the disclosure. The vehicle is equipped with one or more imagesensors 202, which are configured to receive images of a vehicleoccupant. The vehicle may be equipped with one or more processors 203,which are configured to receive image data corresponding to the one ormore images from the image sensor 202, and to determine from the imagedata a gaze direction of a vehicle occupant. As will be described ingreater detail, the gaze direction may be calculated based on at leastone of a calculation of occupant body position, head position, and eyeposition. The one or more processors 203 may be further configured toselect a display location corresponding to the determined gaze directionof the vehicle occupant. The one or more processors 203 may be furtherconfigured to control an image display device to display a visual tag atthe selected display location. The vehicle may further include an imagedisplay device 204, which is configured to display the visual tag at theselected display location according to the one or more processors. Thevisual display device 204 may be any device that is capable ofdisplaying a visual tag. The visual display device 204 may include oneor more projectors to project a visual tag on a surface of or within thevehicle. The visual display device 204 may include one or more screens,configured to display the visual tag at the selected location. In thiscase, the image sensor 202 receives images indicating that theoccupant's gaze is directed to a top left area of the vehicle interior.The one or more processors 203 calculate the gaze based on the imagedata, using at least one of occupant body position, occupant headposition, and occupant eye position. Upon determining a direction ofgaze, the one or more processors map the gaze to an interior point orregion of the vehicle and control the image display device 204 toproject a visual tag 208 at the mapped region. In this case, the visualtag 208 is depicted as a human shape. The visual tag may be of anydesign whatsoever. According to an aspect of the disclosure, the visualtag may be given a human or human-similar shape, which may engenderfeelings of trust or relatability within a human occupant. The visualtag may be given a shape including a face, such as with two eyes and themouth, such that the eyes may be directed to appear to look at one ormore regions of the vehicle, or to make eye contact with the vehicleoccupant.

FIG. 3 depicts a vehicle visual tag system according to a second aspectof the disclosure. As autonomous vehicles develop, and as the need forat least intermittent human control of the vehicle diminishes,autonomous vehicle occupants may conceivably relinquish all traditionaldriving responsibilities, such as acceleration and steering.Accordingly, it is anticipated that some or all occupants may sit in aconfiguration other than the traditional forward-facing vehicleconfiguration. FIG. 3 depicts an alternative seating arrangement,wherein one or more occupants may sit in a round 302. In this case, thenecessity to direct attention in the direction of travel may bediminished, which may permit introduction of one or more occupant focalareas 304 within the vehicle. In this case, focal area 304 may be animage display device, such that the visual tag may be projected from thefocal area 304 to a desired location within the vehicle. The focal area304 may be a physical embodiment of a visual tag, such as an embodiedagent.

FIG. 4 depicts a vehicle visual tag system according to a third aspectof the disclosure. The vehicle may be equipped with one or more imagesensors 401, which are configured to obtain one or more images of a useroccupant. One or more processors 402 within the vehicle may beconfigured to receive the images from the one or more image sensors 401and to determine from the received images an occupant gaze. Based on theoccupant gaze, the one or more processors may select a display locationcorresponding to the determined gaze location of the vehicle occupant.The vehicle may be equipped with a plurality of image display devices,which are configured as one or more screens 403-409. The one or moreprocessors may be configured to select a screen corresponding to thedetermined gaze location, and to control the selected screen to displaythe visual tag. That is, rather than projecting the visual tag on apredetermined location, the one or more processors may select a screenat or within a vicinity of the desired location, and to cause theselected screen to display the visual tag.

FIG. 5 depicts a vehicle visual tag system according to a fourth aspectof the disclosure. In this case, the vehicle is equipped with one ormore image sensors 502, which are configured to obtain images of avehicle occupant. The system further includes one or more processors504, which are configured to determine an occupant gaze from the one ormore images. In this case, rather than projecting or displaying an imageof a visual tag, the vehicle may include a physical embodied agent 506,which may be, for example, a three-dimensional representation of avisual tag. According to one aspect of the disclosure, the physicalembodied agent 506 may resemble a human shape. The physical embodiedagent may include at least one of a head, a torso, upper extremities,lower extremities, or any combination thereof. The physical embodiedagent 506 may be located anywhere within the motor vehicle. The physicalembodied agent 506 may be designed as a focal point for paralinguisticgaze of one or more occupants, such that an occupant's gaze is directedto, or conditioned by, the physical embodied agent 506 duringcommunication.

An occupant may directly gaze in a direction other than the physicalembodied agent 506, such as to an area of the vehicle, or to an areaoutside of the vehicle. Element 508 depicts a gaze of an occupant beingdirected to a region outside of the vehicle. The physical embodied agent506 is within a line of sight 510 of the occupant, or the occupantotherwise expresses a belief that the agent to have access to thedirection of gaze. The physical embodied agent may be configured toreceive images corresponding to a vicinity of the vehicle 512 and tocorrelate the occupant gaze 508 with an object in the vicinity of thevehicle 512. Accordingly, the physical embodied agent 506 may beequipped to recognize an occupant's point of focus outside of thevehicle. The physical embodied agent 506 may be equipped with one ormore image sensors on or within the physical embodied agent, or thephysical embodied agent may receive information or data from one or moreimage sensors external to the physical embodied agent. Similarly, anyprocessing necessary for the physical embodied agent may be performed byone or more processors located within or external to the physicalembodied agent.

FIG. 6 depicts a physical embodied agent's correlation of an occupant'sgaze with an occupant's gesture. In situations where the physicalembodied agent may recognize a gaze of a vehicle occupant, thedetermined gaze may be associated with a direction or a vector, whichmay be extended from the occupant in a specified direction, to provideinformation about an object or point of focus of the occupant. Withoutfurther information, however, it may be difficult or impossible todiscern the specific object of focus of the occupant, as the object maytheoretically be any object along a vector or line extending outwardfrom the occupant. Additional information about an occupant's attentionor visual focus may be derived from an occupant's gesture.Disambiguation may be multimodal; for example, language, gesture,posture, and/or gaze may all work together to help a machineintelligence to understand how to ground a passenger's reference. InFIG. 6, an occupant's gesture 610, such as, but not limited to, anextended arm pointing in a direction, is depicted along with adetermined gaze 612. Two vehicles 606 and 608 are depicted as beingoutside of the occupants of vehicle, as indicated by a line ofdemarcation 604 differentiating between a region inside the vehicle anda region outside of the vehicle. It is anticipated that a human occupantmay instruct the autonomous vehicle to perform an activity, such thatthe instruction may be ambiguous. For demonstrative purposes, a specificexample may be an occupant instructing an autonomous vehicle in avicinity of two vehicles 606 and 608 to park “next to the car.” A verbalcue alone (such as a verbal instruction to park “next to the car”) mayprovide insufficient information for the autonomous vehicle, as multiplecars may be present, and therefore it cannot be determined which car ismeant by the occupant. The ability to receive and understand a visualcue alongside a verbal cue, however, may allow the intelligent agent tomuch more accurately understand the occupant's intentions orinstructions. For example, the occupant's gaze (a visual cue) mayprovide additional clues as to the object of the occupant's attention.In this case, the physical embodied agent 602 may detect a first vector612 associated with an occupant's gaze and a second vector 610associated with an occupant's gesture. The two vectors may be evaluatedfor a point of intersection, which may indicate an area of theoccupant's attention. This is not limited to objects or regions outsideof the vehicle but may also include objects or regions within thevehicle as well. That is, the occupant point or direct attention to oneor more objects or regions within the vehicle, and the intelligent agentmay be configured to identify from the occupant's gestures and/or gazethe object or area to which the occupant's attention is directed.Moreover, the detection of an occupant's gaze pay permit the embodiedagent to engage in shared focus with the occupant as a communicativedevice.

FIG. 7 depicts an additional configuration of the first aspect of thedisclosure. In this case, the vehicle occupant 702 is present withattention drawn to a region of the vehicle. One or more image sensors704 receive images of the vehicle occupant 702. The images aretransferred to one or more processors 706, which determine from theimages the occupant's gaze. The one or more processors 706 map theoccupant's gaze to an area or point of the vehicle and control aprojector 708 to project a visual tag 710 on the area corresponding withthe occupant's gaze. By projecting on that area, it becomes unnecessaryfor the vehicle occupant to turn the occupant's gaze elsewhere tocommunicate with the autonomous vehicle while having a vehicle focalpoint.

FIG. 8 depicts an additional configuration of the second aspect of thedisclosure. In this case, a vehicle occupant 802 is within a vehiclewhile turning the occupant's gaze toward a focal point. The one or moreimage sensors 804 receive images of the vehicle occupant 802 andtransfer the images to the one or more processors 806. The one or moreprocessors 806 derive from the image data the occupant's gaze, which ismapped to a point or area of the vehicle. A screen or displaycorresponding to the mapped point or area is selected as a screen ordisplay on which the visual tag is to be displayed. The one or moreprocessors 806 control the selected display 808 to display the visualtag. The vehicle may be equipped with one or more displays, which may belocated anywhere throughout the vehicle the displays may be multipledistinct displays; portions of one or more displays; flat displays;curved displays; bendable displays; or otherwise.

FIG. 9 depicts an additional configuration of the third aspect of thedisclosure. The vehicle occupant 902 is within a vehicle while turningthe occupant's gaze toward a focal point. In this case, the focal pointis the physical embodied object. The one or more image sensors 904 areconfigured to receive images of the vehicle occupant 902 and to transfersaid images or corresponding image data to one or more processors 906.The one or more processors 906 determine from the images or image data auser occupant's gaze. The one or more processors may control thephysical embodied object 908 to perform one or more actionscorresponding to the determined gaze of the occupant. For example, theone or more processors 906 may control the physical embodied object 908to turn its torso, head, or otherwise toward the occupant or gesture tothe occupant.

FIG. 10 depicts a processing workflow. One or more cameras areconfigured to perform image acquisition 1002, and the correspondingimages or image data are transferred along two paths. The first pathleads to eye tracking determination 1004, at which the occupant's eyesmay be detected within the image data. The eye tracking information isevaluated by an eye direction estimator 1006, which determines adirection of the occupant's eyes. Regarding the second path, the imageacquisition information is delivered to a face detection unit 1008,which isolates the occupant's face within the images or image data. Theisolated face information is delivered to a head pose estimator 1010,which determines a head pose configuration relative to a reference pointwithin the system or vehicle, or other head directional informationrelative to a fixed point. The eye direction estimation and head poseestimation information are delivered to the fusion and adaptation module(“FAM”) 1012, which synthesizes the eye direction information and headpose information to determine an occupant gaze.

FIG. 11 shows elements of a gaze calculation. According to one aspect ofthe disclosure, the system described herein may calculate a precise gazeof an occupant. According to another aspect of the disclosure, however,a precise gaze calculation may be unnecessary, and therefore it maysuffice to omit one or more of the six degrees of freedom within thecalculation, such omitting any of movement along an x-axis, a y-axis, az-axis, roll, yaw, and pitch. For example, as described herein, it maybe possible to calculate a gaze of sufficient accuracy using only thehead position and the yaw angle of the eyeballs, as depicted in FIG. 11.In this figure, the dashed lines 1106 and 1108 represent perpendicularplanes slicing through the occupant from a bird's eye perspective. Thesolid arrows 1104 indicate the direction the occupant is actuallylooking. The dashed lines 1102 along with angle ‘m’ indicate the yawangle of the head with respect to a reference point. In this case, thereference point is, for example, 12 o'clock. Similarly, angle ‘n’ is theyaw angle of the eyeball. The sum ‘m+n’ provides the gaze direction inreference to the occupant's reference point. As will be described laterherein, this aggregated angle for gaze determination may be calculatedby the FAM.

FIG. 12 depicts an operation of the FAM block according to an aspect ofthe disclosure. Following the execution of the procedure 1202, binarystate variables ‘x’ and ‘y’ are initialized to zero in the FAM, as shownby 1202 and 1204. These states permit conditional fusion in FAM, whichmay be desired in cases in which the yaw angles for the eyeball and/orthe head may not be available. The yaw angle of the eyeball (i.e. ‘n’)is determined by the gaze estimator and may be calculated within the[−90,90] degree range. Similarly, the yaw angle of the head (i.e. ‘m’)is received from the head pose estimator block estimator, and may alsovary between [−90,90] degrees. At the fusion stage, it is determinedwhether the eyeball yaw angle is available 1208 and whether the head yawangle is available 1210. If the eyeball yaw angle n is available 1210,then x is set to 1, 1212. If the head yaw angle is available 1216, theny is set to 1, 1218. If either ‘m’ or ‘n’ are not available, only theavailable angle is forwarded to the Passenger Interaction Engine (“PIE”)block 1222. If neither of the angles are available, the output value(i.e. ‘q’) is initialized to 360 degrees 1220, which is outside thevalid range for q of −90 to 90 degrees, and thus indicates anunavailability of yaw angles to the PIE block. The FAM block may beconfigured to continuously update angle ‘q’ 1224, or it may calculate qonce per activation, depending on the context of operation. Thisparameter can be made available as an option to the system designers.When the updates discontinue, then procedure ends 1226.

FIG. 13 depicts the PIE according to an aspect of the disclosure. ThePIE block may accept input from the FAM block, the occupant seatingposition related sensor information, and the vehicle's hardwareconfiguration information. Optionally, audio based context informationcan be provided to PIE block in order to enhance its controller dataprovisioning capabilities (i.e. controller data output can be calculatedbased on audio as well as visual information). Upon activation 1302, thePIE block receives an audio prompt or a fused yaw angle availability1304, and acquires the seating position of the occupant 1306A, as wellas the occupant's fused yaw angle 1308 as provided by the FAM block. ThePIE block then identifies which aspect or aspects of the disclosure aresupported by the vehicle, whether projection of a visual tag 1310,displaying of a visual tag on one or more screens 1312, or interactionwith an embodied agent 1314. In the event that no such support isavailable, the procedure ends 1316. The steps corresponding to 1310,1312, and 1314 may operate in parallel, such as in a multi-threadedapplication. In the case of tangible device support, there will be oneoutput device with a fixed location inside the vehicle. Therefore,identification of a target location for animating the agent may beirrelevant. The PIE simply generates the agent's motion vector data 1330and creates a data frame compatible with the subsequent AEC block thatwill activate the tangible device according to the vector dataspecified. The vector data may be updated depending on the context ofthe interaction between the agent and the occupant (as a non-limitingexample for demonstrative purposes, the embodied agent may wave at theoccupant as it turns its head to face the occupant, when the occupantenters the vehicle) 1336.

In the case of dynamic projection and fixed screen embodiments 1312,there may be multiple candidate locations for the display of the visualtag. To manage the possibility of multiple locations, the availablescreens are first identified, and their locations within the vehicle areascertained 1326. The preferred location is then identified based onoccupant's gaze and the seating position 1328. With this information,the tag's fixed screen visual update data is generated 1324, thecombined AEC data frame is generated 1332, and the combined frame isforwarded to the AEC 1334. In the event that dynamic projection issupported 1310, one or more suitable projection locations are identified1318, and the desired projection location is identified from thepotential projection locations based on the occupant's seating positionin gaze 1320, and the visual tag's fixed screen visual update data isgenerated 1322. The combined AEC data frame is generated 1332, and thecombined frame is forwarded to the AEC 1334.

FIG. 14 depicts a procedure for identifying a desired location forprojection or display of the visual tag. In the event that a pluralityof screens is used, any number of screens may be implemented, withoutlimitation. For the sake of convenience, three screens are depicted inFIG. 14; however, the number of screens should not be understood asbeing limited to three. The procedures described herein may beimplemented with one or more occupants sitting at any location withinthe vehicle. In this figure, the vehicle has three screens, screen one1402, screen two 1404, and screen three 1406. The occupant is seatedbehind and between fixed screen to 1404 and fixed screen three 1406, butthe occupant's gaze is directed toward fixed screen one 1402. Based onthe annotations and representations in this figure, the following tablelists the relevant critical parameters and their definitions beforeproviding the calculation details:

Parameter Definition Known/Unknown x Shortest distance between themounted screens Known (part of the and the occupant′s seat vehicle′sbuilt-in attributes) y Separation between the two screens Known (part ofthe vehicle′s built-in attributes) w Distance of occupant seatingposition to Known (part of the nearest screen vehicle′s built-inattributes) q Fused yaw angle for the occupant′s gaze Known (provided byFAM) a Yaw angle when occupant looks directly at Unknown (calculatedbased fixed screen one on seating position) b Yaw angle when occupantlooks directly at Unknown (calculated based fixed screen two on seatingposition) c Yaw angle when occupant looks directly at Unknown(calculated based fixed screen three on seating position) k Days changeeffort angle when current Unknown (calculated based occupant gaze needsto switch to fixed screen on seating position and one current gazeangle) m Days change effort angle when current Unknown (calculated basedoccupant gaze needs to switch to fixed screen on seating position andtwo current gaze angle) n Days change effort angle when current Unknown(calculated based occupant gaze needs to switch to fixed screen onseating position and three current gaze angle)

Angles ‘k’, ‘m’ and ‘n’ may be compared to determine the angle with thesmallest magnitude. The desired screen may be determined as the screenthat will require the least effort for the occupant to shift theoccupant's gaze, given the seating position and the current gazedirection. The number of calculations needed can be reduced through aprocess of elimination based on the sign of angle ‘q’ as indicated inFIG. 14. In the shown example, the sign of ‘q’ is negative, andtherefore all the screens that give a positive ‘q’ when viewed by theoccupant can be eliminated as candidate screens, as the occupant willhave to spend more effort to redirect the occupant's gaze. Therefore,angle ‘n’ can be eliminated as a candidate screen, as fixed screen 3requires more gaze redirection than Fixed Screen 1 or Fixed Screen 2. Asthe next step, the remaining angles ‘k’ and ‘m’ are calculated asfollows.

$\begin{matrix}{a = {{atan}\left( \frac{w + y}{x} \right)}} & (1) \\{b = {{atan}\left( \frac{w}{x} \right)}} & (2) \\{k = {{{q} - a}}} & (3) \\{m = {{{q} - b}}} & (4)\end{matrix}$

Then the selection of the desired screen to display the embodiment canbe made as follows:If k<m, then use fixed screen 1  (5)If k<m, then use fixed screen 1  (6)

The decision for case k=m is determined in exactly the same way as whenangle ‘q’ is not made available by the FAM block (i.e. when q=360degrees) In such cases, the embodiment target device can be selectedbased on the seating position of the occupant alone.

In the case of dynamic projection and fixed screen embodiments, becausethere are likely to be multiple candidate locations for the embodiments,identification of the desired location based on occupant gaze and theseating position will be required. In addition, the FAM may also beconfigured to select a desired location in situations in which the gazeestimation and/or the head pose estimation may not be available, asthese are independently calculated and forwarded to FAM. Such cornercases are handled at the fusion stage in FAM.

FIG. 15 depicts a determination of a preferred display location in theevent that one or more angles between the occupant and a displaylocation are not available. In FIG. 14, Fixed Screen 3 could beeliminated as an option, thereby leaving Fixed Screen 1 and Fixed Screen2 as potential options for a display location. In this situation, FixedScreen 1 or Fixed Screen 2 may be selected as follows:A=√{square root over (x ²+(y+w)²)}  (7)B=√{square root over (x ² +w ²)}  (8)If A>B, then use fixed screen 2  (9)If A<B, then use fixed screen 1  (10)If angle ‘q’ is not available at the activation of PIE block operations(i.e. q=360 was sent to PIE input), then it may be necessary toadditionally consider Eq. 11.C=√{square root over (x ²+(y−w)²)}  (11)The decision process would then be modified as:If min(A,B,C)=A, then used fixed screen 1  (12)If min(A,B,C)=B, then used fixed screen 2  (13)If min(A,B,C)=C, then used fixed screen 3  (14)where min(.) gives the argument with the minimum magnitude.

The above calculations within the PIE block can be repeated to ensuretimely updating of the fixed screen display and the projection of thevisual tag, based on occupant's gaze and the seating position changes asindicated in FIG. 13. Some implementations of the PIE block may permitimproved decision-making, such as where the fusion takes certainocclusions inside the vehicle between the occupant and the candidatescreens/projection locations into account before choosing the targetdisplay device and/or display location. This would require the PIE blockto work in tandem with various object recognition sub-systems within thevehicle. The fusion algorithms disclosed herein can be applied tosupport such extensions.

FIG. 16 depicts the AEC, which activates the visual tag or physicalembodied agent once the PIE block determines the display location andgenerates the required update data, all of which are integrated into anAEC data frame. FIG. 16 shows the operation of AEC, which is able tosupport projection of the visual tag, display of the visual tag on oneor more displays, use of the physical embodied agent, or any combinationthereof. As depicted herein, once the procedure begins 1602, it isdetermined whether a PIE update is available. If no update is available1604, then a loop is created until an update is provided. Once an updateis available 1606, the combined AEC data frame is parsed 1608. It isdetermined which modalities are supported by the vehicle, whetherdynamic projection of the visual tag 1610, displaying on one or morescreens 1622, or use of a physical embodied agent 1632. In the eventthat dynamic projection is available 1610, the target projectionlocation or selected display location 1612 is obtained, and the targetprojection or selected display dimensions are configured 1614. Aprojection data update of the visual tag is updated 1616, and therendered projection output is displayed at the selected location 1618.Once this is completed is analyzed whether any additional PIE updateinformation is available 1620.

In the event that the visual tag is to be displayed on one or more fixedscreens 1622, the target screen locations are determined 1624, as arethe resolutions for one or more of the target screens 1626. An updatedvirtual tag information, if available, is provided for each screen onwhich the virtual tag will be displayed 1628. The virtual tag is thendisplayed on each screen that is selected 1630. It is noted that, asdescribed herein in at least 1630, it may be contemplated that thevisual tag would be displayed on more than one screen. According to oneaspect of the disclosure, a specific screen may be selected fordisplaying the visual tag based on the methods described herein. Asingle location within the single screen may be selected for displayingthe visual tag. According to another aspect of the disclosure however,it may be desirable to display the visual tag on a plurality of screensas described in 1630.

In the event that a physical embodied agent is used 1632, the agent'smotion vector update is obtained 1634, and target movements on the agentare activated 1636.

In addition to the embodiment data updates, the AEC block also receiveshardware related information directly from PIE, such as targetprojection dimensions, locations, and screen resolutions. According toanother aspect of the disclosure, such information may also be providedto the AEC directly from other sub-systems in the vehicle, as this maybe the preferred implementation for a vehicle manufacturer. The AECblock may also monitor the output of the PIE block to ensure that newupdates are promptly applied to relevant devices to foster a smooth userexperience.

FIG. 17 depicts a vehicle tag system including one or more image sensors1702, configured to receive one or more images of a vehicle occupant;one or more processors 1704, configured to determine a gaze direction ofthe vehicle occupant from the one or more images; select a displaylocation corresponding to the determined gaze direction; and control animage display device 1706 to display a tag at the display location; theimage display device 1706, configured to display the tag at the selecteddisplay location according to the one or more processors.

FIG. 18 depicts a vehicle embodied agent system including a physicalembodied agent 1801; one or more image sensors 1802, configured toreceive one or more images of a vehicle occupant; and one or moreprocessors 1804, configured to determine a gaze direction of the vehicleoccupant from the one or more images; and control the physical embodiedagent to execute a routine according to the determined gaze direction.The one or more image sensors 1802 may be located independently of thephysical embodied agent 1801, or may be included within the physicalembodied agent 1801. Similarly, the one or more processors 1804 may belocated separately from the physical embodied agent 1801, or they may belocated as part of the physical embodied agent 1801.

FIG. 19 depicts a method of displaying a tag including receiving one ormore images of a vehicle occupant 1902; determining a gaze direction ofthe vehicle occupant from the one or more images 1904; selecting adisplay location corresponding to the determined gaze direction 1906;and controlling an image display device to display a tag at the displaylocation 1908.

As autonomous vehicles are developed, it is becoming more important forautonomous vehicles to develop the capability of understanding theintentions of vehicle occupants. Some intentions are simplistic and thusreadily understandable. For example, it is a relatively straightforwardtask for an occupant to inform the vehicle of a desired destination atthe beginning of travel. For example, an occupant may instruct a vehicleto travel to a specific destination or to use a specific route, whichcan be achieved with a simple verbal command. This interaction mayrequire only speech recognition and an ability to cross-referencelocations detected from the speech with actual geographical locations,such as being able to recognize a destination, and then identify thedestination and a route thereto in a database. However, someinteractions between occupants and the vehicle are more challenging,such as spontaneous ad hoc interactions, for example, as with en routemodifications due to traffic problems or changes in the desireddestination. Such situations require more nuanced conversations that maybe beyond the capabilities of even the most robust speech to textsystems. This may be due at least to the fact that information relatedto the occupant's intention is contained in modalities outside of thetext itself, such as the expressions and gestures of the occupant.

The best results may be obtained by developing autonomous vehicles thatare able to understand communication in the normal ways that humanscommunicate with one another. If for no other reason, this may owe tothe fact that occupants can be assumed to have the most experiencecommunicating with other persons. This may go beyond the simple text ofa sentence, and include additional resources such as tone of voice,gesture, features of the environment, and emotional displays (bothfeigned and sincere). Removing any of the non-lexical features from theinput to an autonomous vehicle agent may carry an unnecessary cost.

Devices that are sufficiently context aware can be a part of anautonomous vehicle system such that it can support fully naturalcommunication. Such a system may have the ability to recognize speech,to identify the speaker, and to recognize emotional timbre, etc. Theinput must also include vision to support the use of gesture and otheraspects of contextual awareness (e.g., seeing a phone in the occupant'shand or the building toward which the occupant is pointing). Such visionmay be used as described herein in conjunction with speech todisambiguate the spoken reference.

The one or more image sensors may be any kind of image sensors, whichare capable of obtaining an image of the vehicle occupant or occupants,without limitation. Furthermore, and depending on the desiredconfiguration, the one or more image sensors may be configured to obtainimages of one or more regions and/or objects in a vicinity of a vehicle.Any one or more image sensors may be individually capable of receivingimages from both inside the vehicle and outside the vehicle, or they mayreceive images from either inside the vehicle or outside the vehicle andmay deliver said images to the one or more processors for synthesisany/or any further processing. The images sensors may include, forexample, a still camera, a video camera, a depth camera, an infraredcamera, lidar, or radar. The one or more image sensors may be located atany position or combination of positions within the vehicle. Where aplurality of image sensors are used, the image sensors may be locatedgenerally within a single place on the vehicle, or may be located inmultiple places, so as to obtain different vantages of one or morevehicle occupants. Where a plurality of image sensors are used, the datafrom the image sensors may be assessed separately from each imagesensor, or the image data may be aggregated to form a singular image,which is then assessed by the one or more processors. The dataaggregation may occur given any known image combining or imageaggregation technique.

The one or more processors may be any type of processor whatsoever,which are capable of performing the tasks described herein. The one ormore processors may be located anywhere within or external to thevehicle, depending on the implementation. Where the one or moreprocessors are located within the vehicle, they may be located withinthe embodied agent, within the image display device, or anywhere elsewithin the vehicle. Where the one or more processors are locatedexternal to the vehicle, they may be located in any place whatsoever,and the vehicle may be configured to transmit data such as image sensordata to the one or more processors, and to receive from the one or moreprocessors information for projecting or displaying the visual tag, orinformation for execution of the embodied agent.

The gaze direction may be determined with respect to any location.According to one aspect of the disclosure, one or more fixed locationswithin the vehicle may be dedicated as a fixed location relative towhich the gaze is determined. Determining the gaze may include anycombination of determining a position of a vehicle occupant, determininga body orientation of the vehicle occupant, determining a headorientation of the vehicle occupant, and/or determining the eyeorientation of the vehicle occupant.

The display location may be any surface on or within the vehicle. Thedisplay location may include, without limitation, a dashboard, awindshield, a window, a panel, a wall, a door, a ceiling, or any otherportion of the vehicle.

The visual tag may be displayed or projected at any size or resolution.According to one aspect of the disclosure, the size and/or resolution ofthe visual tag may be selected based on any of a distance between thevehicle occupant and the display location, a projection quality of thedisplay location, or any other feature.

As described herein, the gaze direction may be calculated from at leastone of an eye position, a head position, a body position, a physicallocation of a vehicle occupant, or any combination thereof.

According to one aspect of the disclosure, the calculations for gazedirection may be simplified by calculating, where possible, a head yawand eye yaw. This may be at least because the possibilities fordisplaying or projecting the visual tag may be greater horizontally thanvertically. Otherwise stated, there may be many surfaces of roughly thesame height, or roughly along the same horizontal plane, on which thevisual tag may be projected or displayed; however, depending on thevehicle configuration, there may be fewer surfaces along a verticalplane on which the tag may be displayed. As such, it may be sufficientto calculate a head yaw and eye yaw, which may provide information abouta location within the given horizontal plane at which the visual tag mayideally be projected or displayed.

Alternatively, any or all of the six degrees of freedom may becalculated with respect to any combination of eye position, headposition, body position, or any combination thereof, and thisinformation may be used to select a display location.

According to one aspect of the disclosure, the image display device maybe a projector, which is configured to project the visual tag onto asurface. The projector may project the visual tag at any size orresolution desired. The one or more processors may be configured tomodify the projection information to account for anything other than aprojection surface that is perpendicular to the projection direction.That is, where the projection surface is flat but angled such that it isnot perpendicular to the line of projection, the one or more processorsmay be configured to modify the projection data such that the projectionon the angled projection surface appears to the vehicle occupant as anunmodified visual tag. Where the projection surface is uneven, such aswith a curved surface, the one or more processors may be configured tomodify the projection data such that the projection on the curved oruneven surface appears to the vehicle occupant as an unmodified visualtag. This modification may be performed using any known methodincluding, but not limited to, any known projection mapping technique.

According to another aspect of the disclosure, the image display devicemay be a display, such as a screen that is configured to display one ormore images. The display may be located at any position within thevehicle including, but not limited to, within a central console, on arear surface of a seat, on a wall, on a ceiling, or anywhere else withinthe vehicle whatsoever. A plurality of displays may be arranged withinthe vehicle, such that selecting the display location includes selectinga particular display on which the visual tag will be displayed.Furthermore, selecting a display location may include selecting aparticular point or points on the selected display on which the visualtag will be displayed. Where desired, a plurality of displays may beselected to jointly display the visual tag. Where this occurs, thevisual tag may be repeatedly displayed on the plurality of displays,such that an identical or nearly identical image is found on each of thedisplays, or the visual tag may be divided, such that a portion of thevisual tag is displayed on a first display, and a portion of the visualtag displayed on another display. This technique may incorporate anynumber of displays, as desired. The displays may be any kind of displayswhatsoever, whether LCD, OLED, flat, curved, flexible, or otherwise.

According to another aspect of the disclosure, the system describedherein may be equipped with the ability to coordinate a movement of anoccupant within a vicinity of the vehicle. In normal human-to-humaninteraction, it is common to point or otherwise gesticulate toward anobject that is the subject of a conversation. If communication between ahuman and a vehicle is limited to entering data into an interface, suchas a screen or control, or by using voice cues devoid of physicalinformation, such gestures cannot be included in communication with thevehicle.

It is thus anticipated that the vehicle may derive from the one or moreimage sensors a direction of a gesture of an occupant, such as adirection of pointing. Coordinates or headings for the direction may beobtained, which may be a vector, with either a calculated orpredetermined length, a line extending in the direction of pointing, orotherwise. The vehicle may be equipped with one or more image sensors,which are configured to obtain images from the vicinity of the vehicle.The one or more processors may be configured to correlate a direction ofthe pointing using the vector, line, or otherwise, with the image dataregarding the vicinity of the vehicle. Using this technique, the vehiclemay be able to correlate a gesture with an object, such as pointing to avehicle, a parking spot, a street, or otherwise. The one or moreprocessors may be configured to incorporate this data into any availablespeech recognition, such that the vehicle may be assisted inunderstanding the occupant's intentions. For example, an occupantstating “Park over there” while pointing to a parking space may haveprovided the vehicle within insufficient information to complete thedesired task, if only text or only gesture are considered; however, ifthe vehicle is able to understand both text and gesture, the vehicle maybe equipped to locate the intended parking spot and carry out thecommand to park at that location.

Similarly, the vehicle may be equipped to correlate an occupant's gazewith an external location. This may be carried out in much the same waythat the vehicle correlates a gesture or pointing with an exteriorlocation. The vehicle may use any of the techniques described herein toidentify a direction of an occupant's gaze, and may extend the directionof that gaze outward beyond the vehicle, until its intersection with anobject is detected by one or more image sensors obtaining image dataabout the vicinity of the vehicle.

Using the above-described techniques, the vehicle may also correlate anoccupant's gaze and an occupant's gesture to identify an externallocation. In this case, a direction of the occupant's gaze and adirection of the occupant's gesture may be correlated with informationabout the vicinity of the vehicle to identify an object or locationexternal to the vehicle. Because an occupant's gesture, such as with anupper extremity, originates from a different physical place than anoccupant's gaze, vectors and/or lines extending in the direction of thegaze and pointing may ultimately intersect. This point of intersectionmay correspond with an intended object or location. A vector or linecorresponding to a determined gaze, and a vector or line correspondingto a determined gesture, may be in two different planes and would thusnot intersect. According to one aspect of the disclosure, one or both ofthe vectors and/or lines may be modified such that the vector or linecorresponding to the determined gaze and the vector or linecorresponding to the determined gesture may be modified to be within thesame plane and thus intersect.

According to another aspect of the disclosure, the vehicle may beequipped with a physical embodied agent. The physical embodied agent maybe a physical, three-dimensional representation of the vehicle and/orits artificial intelligence. The physical embodied agent may take anyform whatsoever, including a three-dimensional variety of any formdescribed herein with respect to the visual tag.

Upon determining an occupant's gaze, the one or more processors maycontrol the physical embodied agent to execute a routine. The routinemay be any routine whatsoever. Non-limiting examples of said routine mayinclude turning the physical embodied agent to face the occupant,causing the physical embodied agent to perform one or more gestures,causing the head of the physical embodied agent to turn toward theoccupant, or causing the eyes of the physical embodied agent to turntoward the occupant, or toward a third location to enable a shared focuswith the occupant.

The physical embodied agent may include one or more joints or movable orbendable portions. The physical embodied agent may be configured toadjust the one or more joints or movable or bendable portions so as tochange position and/or posture within the vehicle, or to gesticulate inany way. The joints may be configured in a manner such as to mimic orapproximate human movement, or in any other manner desired.

According to one aspect of the disclosure, the projector may be ahologram projector, and the virtual agent may be projected in the formof a hologram. In this case, the one or more image sensors will detectthe head position and an eye position as described herein, and thedetected head position and an eye position will be mapped to an area ofthe vehicle. The one or more processors may be configured to cause thehologram projector to project the hologram of the virtual agent in anarea corresponding to the mapped area, as determined from the calculatedhead position and an eye position. Where the hologram is configured as athree dimensional hologram, the hologram of the virtual agent may berotated, such that the virtual agent appears to face the vehicleoccupant. This may include rotating the virtual agent any number ofdegrees. Alternatively, the virtual agent may be rotated to appear tolook at or point to an object in a direction other than the vehicleoccupant.

Although the methods and procedures described herein have been detailedwith respect to a vehicle, such as an autonomous vehicle, the methodsand procedures described herein may also be applied to situations otherthan in the vehicle, such as in buildings, in public spaces, orotherwise. The one or more image sensors may be placed in any locationto obtain image data of a person, and the person's head position and aneye position may be determined in the manner described herein. Thecombined head position and an eye position may be used to determine agaze of the person, and a location may be selected for displaying anintelligent agent based on the determined gaze. Whereas this has beendescribed with respect to a vehicle, the person may be located in anyplace whatsoever, whether in a building, or outside. Where a physicalembodied agent is used, the physical embodied agent may be configured toperform one or more activities or trigger one or more activities basedon the determined gaze in the manner described with respect to thephysical embodied agent in the vehicle herein. Possible nonlimitingexamples of extra-vehicular places where this technology may be usedinclude stores, commercial establishments, schools, public buildings,private buildings, public outdoor spaces, private outdoor spaces, andhomes. In a multiple vehicle-occupant scenario, the projection and/ordisplay of the embodied agent may be performed such that multipleembodied agents are projected or displayed. That is, an embodied agentmay be displayed for each of multiple vehicle occupants. Thus, the oneor more processors may be configured to display, control, animate, orotherwise operate multiple embodied agents concurrently. In somescenarios, such as in the event that a projection mapping technology isused with multiple occupants, it may be undesirable to use a singleembodied agent for two or more vehicle occupants, due at least toprojection mapping's dependence upon viewing angle.

Alternatively, in a multiple vehicle-occupant scenario, it may bedesirable to select a single location for projection of the embodiedagent. For at least the reasons stated above, and particularly where aprojection mapping algorithm is employed, selection of a single locationfor multiple occupants may create additional challenges, since effectiveexecution of the projection mapping technology may be dependent on aviewing angles of the one or more occupants. This is, if an idealprojection mapping is implemented for a specific viewing angle of asingle occupant, the resulting projection may appear skewed or otherwisedistorted from a viewing angle of another occupant. According to oneaspect of the disclosure, in the event that a plurality of occupants arepresent in the vehicle, and a single projection of the embodied agentfor two or more occupants is desired, a single projection location maybe selected as a compromise between two or more ideal projection angles.The single projection location may be selected to provide an acceptablerepresentation of the embodied agent for the two or more occupants. Theprojection location may be selected based on at least one of theoccupants' gazes, the occupants' viewing angles of the projection, acalculated distortion of the projection in light of the occupants'viewing angles, and any combination thereof.

According to another aspect of the disclosure, and in amultiple-occupant situation, the one or more processors may beconfigured to determine from images of two or more occupants a jointoccupant gaze. In conversation, humans may engage in joint gaze, whereinthe humans direct their gaze to a common point of focus, such as lookingat a single object, at a single region, or a common space. This mayoccur, for example, when humans pass an object from one human toanother, as they may direct their gaze from a face of a conversationpartner to the object that is being passed. The one or more processorsmay be configured to determine a location of joint occupant gaze. Wherean area of joint occupant gaze is present, the one or more processorsmay be configured to configure the embodied agent to also appear todirect its attention to the area of joint gaze. That is, the body, head,and/or eyes may appear to turn toward the area of join occupant gaze.

The following examples pertain to various aspects of the Disclosure:

In Example 1, a vehicle tag system is disclosed comprising one or moreimage sensors, configured to receive one or more images of a vehicleoccupant; one or more processors, configured to determine a gazedirection of the vehicle occupant from the one or more images; select adisplay location corresponding to the determined gaze direction; andcontrol an image display device to display a tag at the displaylocation; the image display device, configured to display the tag at theselected display location according to the one or more processors.

In Example 2, the vehicle tag system of Example 1 is disclosed, whereinthe one or more processors are configured to determine the gazedirection of the vehicle occupant from the one or more images bydetermining from the one or more images a head position and an eyeposition of the vehicle occupant.

In Example 3, the vehicle tag system of Example 1 is disclosed, whereinthe one or more processors are configured to determine from the one ormore images a head yaw and an eye yaw, and wherein determining gazedirection comprises combining the determined head yaw and eye yaw.

In Example 4, the vehicle tag system of any one of Examples 1 to 3 isdisclosed, wherein the image display device is a projector is disclosed,wherein controlling the image display device to display the tagcomprises controlling the projector to project the tag onto the selecteddisplay location.

In Example 5, the vehicle tag system of Example 4 is disclosed, furthercomprising determining distances between the projector and a pluralityof points on the selected display location, and modifying the tagaccording to a projection mapping algorithm based on the determineddistances.

In Example 6, the vehicle tag system of any one of Examples 1 to 3 isdisclosed, wherein the image display device is a display, and whereincontrolling the display to display the tag at the selected displaylocation comprises determining a location on the display thatcorresponds to the determined gaze direction and controlling the displayto display the tag at the determined location.

In Example 7, the vehicle tag system of Example 6 is disclosed, whereinthe display is one of a flat screen display, a curved screen display, ora flexible screen display.

In Example 8, the vehicle tag system of any one of Examples 1 to 3 isdisclosed, further comprising a plurality of image display deviceswithin the vehicle is disclosed, wherein the plurality of image displaydevices are displays, and wherein selecting the display locationcomprises selecting one of the plurality of displays corresponding tothe determined gaze direction, and wherein controlling the image displaydevice to display the tag at the selected display location comprisescontrolling the selected display to display the tag.

In Example 9, the vehicle tag system of Example 8 is disclosed, whereinthe plurality of displays comprise at least a flat display, a curveddisplay, a flexible display, of any combination thereof.

In Example 10, the vehicle tag system of any one of Examples 1 to 9 isdisclosed, wherein the tag is an embodied agent.

In Example 11, the vehicle tag system of any one of Examples 1 to 9 isdisclosed, wherein the tag is an avatar.

In Example 12, the vehicle tag system of any one of Examples 1 to 9 isdisclosed, wherein the tag is a physical representation of an artificialintelligence.

In Example 13, the vehicle tag system of any one of Examples 1 to 12 isdisclosed, wherein, if the vehicle comprises a plurality of vehicleoccupants, the one or more processors are configured to identify theplurality of vehicle occupants in the one or more images; select one ofthe vehicle occupants according to a selection instruction; determine agaze direction of the selected vehicle occupant from the one or moreimages; select a display location corresponding to the determined gazedirection; and control the image display device to display a tag at thedisplay location.

In Example 14, the vehicle tag system of any one of Examples 1 to 13 isdisclosed, further comprising one or more sensors, configured to receiveinformation of a vicinity of the vehicle; and wherein the one or moreprocessors are further configured to select a point in the vicinity ofthe vehicle corresponding to a visual cue of the vehicle occupant.

In Example 15, the vehicle tag system of Example 14 is disclosed,wherein the visual cue is a gaze direction of the occupant.

In Example 16, the vehicle tag system of Example 14 or 15 is disclosed,wherein the visual cue is a direction of an upper extremity of anoccupant, and wherein the one or more processors are further configuredto detect from the one or more images an upper extremity of an occupant;and determine the visual cue according to a direction of the upperextremity.

In Example 17, the vehicle tag system of Example 16 is disclosed,further comprising selecting a point in the vicinity of the vehiclecorresponding to both the gaze direction and the direction of the upperextremity.

In Example 18, a vehicle embodied agent system is disclosed comprising:a physical embodied agent; one or more image sensors, configured toreceive one or more images of a vehicle occupant; and one or moreprocessors, configured to determine a gaze direction of the vehicleoccupant from the one or more images; and control the physical embodiedagent to execute a routine according to the determined gaze direction.

In Example 19, the vehicle embodied agent system of Example 18 isdisclosed, wherein the one or more processors are configured todetermine the gaze direction of the vehicle occupant from the one ormore images by determining from the one or more images a head positionand an eye position of the vehicle occupant.

In Example 20, the vehicle embodied agent system of Example 18 isdisclosed, wherein the one or more processors are configured todetermine from the one or more images a head yaw and an eye yaw, andwherein determining gaze direction comprises combining the determinedhead yaw and eye yaw.

In Example 21, the vehicle embodied agent system of Example 18 isdisclosed, wherein the one or more processors are configured todetermine the gaze direction of the vehicle occupant by determining fromthe one or more images a head position and an eye position of thevehicle occupant.

In Example 22, the vehicle embodied agent system of any one of Examples18 to 21 is disclosed, wherein the physical embodied agent is configuredwith a moveable portion comprising at least one of a moveable head;moveable eyes; a moveable torso; one or more moveable appendages; or anycombination thereof.

In Example 23, the vehicle embodied agent system of Example 22 isdisclosed, wherein executing a routine comprises moving a moveableportion.

In Example 24, the vehicle embodied agent system of Example 22 or 23 isdisclosed, wherein executing a routine comprises moving a moveableportion such that the embodied agent's eyes are positioned to face theoccupant.

In Example 25, the vehicle embodied agent system of any one of Examples18 to 24 is disclosed, further comprising one or more sensors,configured to receive information of a vicinity of the vehicle; andwherein executing the routine comprises selecting a point in thevicinity of the vehicle corresponding to a visual cue of the vehicleoccupant.

In Example 26, the vehicle embodied agent system of Example 25 isdisclosed, wherein the visual cue is a gaze direction of the occupant.

In Example 27, the vehicle embodied agent system of Example 25 or 26 isdisclosed, wherein the visual cue is a direction of an upper extremityof an occupant, and wherein the one or more processors are furtherconfigured to detect from the one or more images an upper extremity ofan occupant; and determine the visual cue according to a direction ofthe upper extremity.

In Example 28, the vehicle tag system of Example 27 is disclosed,further comprising selecting a point in the vicinity of the vehiclecorresponding to both the gaze direction and the direction of the upperextremity.

In Example 29, the vehicle embodied agent system of any one of Examples22 to 28 is disclosed, wherein the one or more processors are furtherconfigured to control the moveable portion to move based on thedetermined gaze direction.

In Example 30, the vehicle embodied agent system of any one of Examples22 to 29 is disclosed, wherein the one or more processors are furtherconfigured to control the moveable portion to move based on the upperextremity position.

In Example 31, the vehicle embodied agent system of any one of Examples22 to 30 is disclosed, wherein the one or more processors are furtherconfigured to control the moveable portion to move based on both thedetermined gaze direction and the determined upper extremity position.

In Example 32, the vehicle embodied agent system of any one of Examples22 to 30 is disclosed, wherein the one or more processors are furtherconfigured to control the moveable portion to move based on a movementof the occupant.

In Example 33, the vehicle embodied agent system of any one of Examples18 to 32 is disclosed, wherein the physical embodied agent is a robot.

In Example 34, the vehicle embodied agent system of any one of Examples18 to 32 is disclosed, wherein the physical embodied agent comprises amechanical likeness of a human being.

In Example 35, a method of displaying a tag is disclosed comprisingreceiving one or more images of a vehicle occupant; determining a gazedirection of the vehicle occupant from the one or more images; selectinga display location corresponding to the determined gaze direction; andcontrolling an image display device to display a tag at the displaylocation.

In Example 36, the method of displaying a tag of Example 35 isdisclosed, wherein determining the gaze direction of the vehicleoccupant from the one or more images comprises determining from the oneor more images a head position and an eye position of the vehicleoccupant.

In Example 37, the method of displaying a tag of Example 35 isdisclosed, further comprising determining from the one or more images ahead yaw and an eye yaw, and wherein determining gaze directioncomprises combining the determined head yaw and eye yaw.

In Example 38, the method of displaying a tag of any one of Examples 35to 37 is disclosed, wherein controlling the image display device todisplay the tag comprises controlling a projector to project the tagonto the selected display location.

In Example 39, the method of displaying a tag of Example 38 isdisclosed, further comprising determining distances between theprojector and a plurality of points on the selected display location,and modifying the tag according to a projection mapping algorithm basedon the determined distances.

In Example 40, the method of displaying a tag of any one of Examples 35to 39 is disclosed, wherein controlling the image display device todisplay the tag at the selected display location comprises determining alocation on a display that corresponds to the determined gaze directionand controlling the display to display the tag at the determinedlocation.

In Example 41, the method of displaying a tag of any one of Examples 35to 40 is disclosed, wherein selecting the display location comprisesselecting one of a plurality of displays corresponding to the determinedgaze direction, and wherein controlling the image display device todisplay the tag at the selected display location comprises controllingthe selected display to display the tag.

In Example 42, the method of displaying a tag of any one of Examples 35to 41 is disclosed, wherein the tag is an embodied agent.

In Example 43, the method of displaying a tag of any one of Examples 35to 41 is disclosed, wherein the tag is an avatar.

In Example 44, the method of displaying a tag of any one of Examples 35to 43 is disclosed, wherein the tag is a physical representation of anartificial intelligence.

In Example 45, the method of displaying a tag of any one of Examples 35to 44 is disclosed, wherein, if the vehicle comprises a plurality ofvehicle occupants, the one or more processors are configured to identifythe plurality of vehicle occupants in the one or more images; select oneof the vehicle occupants according to a selection instruction; determinea gaze direction of the selected vehicle occupant from the one or moreimages; select a display location corresponding to the determined gazedirection; and control the image display device to display a tag at thedisplay location.

In Example 46, the method of displaying a tag of any one of Examples 35to 45 is disclosed, further comprising receiving from one or moresensors information of a vicinity of the vehicle; and selecting a pointin the vicinity of the vehicle corresponding to a visual cue of thevehicle occupant.

In Example 47, the method of displaying a tag of Example 46 isdisclosed, wherein the visual cue is a gaze direction of the occupant.

In Example 48, the method of displaying a tag of Example 46 or 47 isdisclosed, wherein the visual cue is a direction of an upper extremityof an occupant; further comprising detecting from the one or more imagesan upper extremity of an occupant; and determining the visual cueaccording to a direction of the upper extremity.

In Example 49, the method of displaying a tag of Example 48 isdisclosed, further comprising selecting a point in the vicinity of thevehicle corresponding to both the gaze direction and the direction ofthe upper extremity.

In Example 50, a method of operating a physical embodied agent isdisclosed comprising receiving from one or more sensors one or moreimages of a vehicle occupant; determining a gaze direction of thevehicle occupant from the one or more images; and controlling thephysical embodied agent to execute a routine according to the determinedgaze direction.

In Example 51, the method of operating a physical embodied agent ofExample 50 is disclosed, further comprising determine the gaze directionof the vehicle occupant from the one or more images by determining fromthe one or more images a head position and an eye position of thevehicle occupant.

In Example 52, the method of operating a physical embodied agent ofExample 50 is disclosed, further comprising determining from the one ormore images a head yaw and an eye yaw, and wherein determining gazedirection comprises combining the determined head yaw and eye yaw.

In Example 53, the method of operating a physical embodied agent ofExample 52 is disclosed, further comprising determining the gazedirection of the vehicle occupant by determining from the one or moreimages a head position and an eye position of the vehicle occupant.

In Example 54, the method of operating a physical embodied agent of anyone of Examples 50 to 53 is disclosed, wherein the physical embodiedagent is configured with a moveable portion comprising at least one of amoveable head; moveable eyes; a moveable torso; one or more moveableappendages; or any combination thereof.

In Example 55, the method of operating a physical embodied agent ofExample 54 is disclosed, wherein executing a routine comprises moving amoveable portion.

In Example 56, the method of operating a physical embodied agent ofExample 54 or 55 is disclosed, wherein executing a routine comprisesmoving a moveable portion such that the embodied agent's eyes arepositioned to face the occupant.

In Example 57, the method of operating a physical embodied agent of anyone of Examples 50 to 56, further receiving from one or more sensorsinformation of a vicinity of the vehicle; and wherein executing theroutine comprises selecting a point in the vicinity of the vehiclecorresponding to a visual cue of the vehicle occupant.

In Example 58, the method of operating a physical embodied agent ofExample 57 is disclosed, wherein the visual cue is a gaze direction ofthe occupant.

In Example 59, the method of operating a physical embodied agent ofExample 57 or 58 is disclosed, wherein the visual cue is a direction ofan upper extremity of an occupant; further comprising detecting from theone or more images an upper extremity of an occupant; and determiningthe visual cue according to a direction of the upper extremity.

In Example 60, the method of operating a physical embodied agent ofExample 59 is disclosed, further comprising selecting a point in thevicinity of the vehicle corresponding to both the gaze direction and thedirection of the upper extremity.

In Example 61, the method of operating a physical embodied agent of anyone of Examples 54 to 60 is disclosed, further comprising controllingthe moveable portion to move based on the determined gaze direction.

In Example 62, the method of operating a physical embodied agent of anyone of Examples 54 to 61 is disclosed, further comprising controllingthe moveable portion to move based on the upper extremity position.

In Example 63, the method of operating a physical embodied agent of anyone of Examples 54 to 62 is disclosed, further comprising controllingthe moveable portion to move based on both the determined gaze directionand the determined upper extremity position.

In Example 64, the method of operating a physical embodied agent of anyone of Examples 54 to 63 is disclosed, further comprising controllingthe moveable portion to move based on a movement of the occupant.

In Example 65, the method of operating a physical embodied agent of anyone of Examples 35 to 64 is disclosed, wherein the physical embodiedagent is a robot.

In Example 66, the method of operating a physical embodied agent of anyone of Examples 35 to 64 is disclosed, wherein the physical embodiedagent comprises a mechanical likeness of a human being.

In Example 67, the method of operating a physical embodied agent ofExample 4 or 5 is disclosed, wherein the projector is a hologramprojector, and the virtual agent is projected as a hologram.

In Example 68, a virtual embodiment display system is disclosedcomprising: one or more image sensors, configured to receive one or moreimages of a vehicle occupant; one or more processors, configured todetermine a gaze direction of the vehicle occupant from the one or moreimages; select a display location corresponding to the determined gazedirection; and control an image display device to display a virtualembodiment of an intelligent agent at the display location; the imagedisplay device, configured to display the virtual embodiment of theintelligent agent at the selected display location according to the oneor more processors.

In Example 69, the virtual embodiment display system of Example 68 isdisclosed, wherein the one or more processors are configured todetermine the gaze direction of the vehicle occupant from the one ormore images by determining from the one or more images a head positionand an eye position of the vehicle occupant.

In Example 70, the virtual embodiment display system of Example 68 isdisclosed, wherein the one or more processors are configured todetermine from the one or more images a head yaw and an eye yaw, andwherein determining gaze direction comprises combining the determinedhead yaw and eye yaw.

In Example 71, the virtual embodiment display system of any one ofExamples 68 to 70 is disclosed, wherein the image display device is aprojector is disclosed, wherein controlling the image display device todisplay the virtual embodiment of the intelligent agent comprisescontrolling the projector to project the virtual embodiment of theintelligent agent onto the selected display location.

In Example 72, the virtual embodiment display system of Example 71 isdisclosed, further comprising determining distances between theprojector and a plurality of points on the selected display location,and modifying the virtual embodiment of the intelligent agent accordingto a projection mapping algorithm based on the determined distances.

In Example 73, the virtual embodiment display system of any one ofExamples 68 to 70 is disclosed, wherein the image display device is adisplay, and wherein controlling the display to display the virtualembodiment of the intelligent agent at the selected display locationcomprises determining a location on the display that corresponds to thedetermined gaze direction and controlling the display to display thevirtual embodiment of the intelligent agent at the determined location.

In Example 74, the virtual embodiment display system of Example 73 isdisclosed, wherein the display is one of a flat screen display, a curvedscreen display, or a flexible screen display.

In Example 75, the virtual embodiment display system of any one ofExamples 68 to 70 is disclosed, further comprising a plurality of imagedisplay devices within the vehicle is disclosed, wherein the pluralityof image display devices are displays, and wherein selecting the displaylocation comprises selecting one of the plurality of displayscorresponding to the determined gaze direction, and wherein controllingthe image display device to display the virtual embodiment of theintelligent agent at the selected display location comprises controllingthe selected display to display the virtual embodiment of theintelligent agent.

In Example 76, the virtual embodiment display system of Example 75 isdisclosed, wherein the plurality of displays comprise at least a flatdisplay, a curved display, a flexible display, of any combinationthereof.

In Example 77, the virtual embodiment display system of any one ofExamples 68 to 76 is disclosed, wherein the virtual embodiment of theintelligent agent is an embodied agent.

In Example 78, the virtual embodiment display system of any one ofExamples 68 to 77 is disclosed, wherein the virtual embodiment of theintelligent agent is an avatar.

In Example 79, the virtual embodiment display system of any one ofExamples 68 to 78 is disclosed, wherein the virtual embodiment of theintelligent agent is a physical representation of an artificialintelligence.

In Example 80, the virtual embodiment display system of any one ofExamples 68 to 79 is disclosed, wherein, if the vehicle comprises aplurality of vehicle occupants, the one or more processors areconfigured to: identify the plurality of vehicle occupants in the one ormore images; select one of the vehicle occupants according to aselection instruction; determine a gaze direction of the selectedvehicle occupant from the one or more images; select a display locationcorresponding to the determined gaze direction; and control the imagedisplay device to display a virtual embodiment of the intelligent agentat the display location.

In Example 81, the virtual embodiment display system of any one ofExamples 68 to 80 is disclosed, further comprising one or more sensors,configured to receive information of a vicinity of the vehicle; andwherein the one or more processors are further configured to select apoint in the vicinity of the vehicle corresponding to a visual cue ofthe vehicle occupant.

In Example 82, the virtual embodiment display system of Example 81 isdisclosed, wherein the visual cue is a gaze direction of the occupant.

In Example 83, the virtual embodiment display system of Example 81 or 82is disclosed, wherein the visual cue is a direction of an upperextremity of an occupant, and wherein the one or more processors arefurther configured to detect from the one or more images an upperextremity of an occupant; and determine the visual cue according to adirection of the upper extremity.

In Example 84, the virtual embodiment display system of Example 83 isdisclosed, further comprising selecting a point in the vicinity of thevehicle corresponding to both the gaze direction and the direction ofthe upper extremity.

In Example 85, a vehicle embodied agent system is disclosed comprising aphysical embodiment of the intelligent agent; one or more image sensors,configured to receive one or more images of a vehicle occupant; and oneor more processors, configured to determine a gaze direction of thevehicle occupant from the one or more images; and control the physicalembodiment of the intelligent agent to execute a routine according tothe determined gaze direction.

In Example 86, the vehicle embodied agent system of Example 85 isdisclosed, wherein the one or more processors are configured todetermine the gaze direction of the vehicle occupant from the one ormore images by determining from the one or more images a head positionand an eye position of the vehicle occupant.

In Example 87, the vehicle embodied agent system of Example 85 isdisclosed, wherein the one or more processors are configured todetermine from the one or more images a head yaw and an eye yaw, andwherein determining gaze direction comprises combining the determinedhead yaw and eye yaw.

In Example 88, the vehicle embodied agent system of Example 85 isdisclosed, wherein the one or more processors are configured todetermine the gaze direction of the vehicle occupant by determining fromthe one or more images a head position and an eye position of thevehicle occupant.

In Example 89, the vehicle embodied agent system of any one of Examples85 to 88 is disclosed, wherein the physical embodiment of theintelligent agent is configured with a moveable portion comprising atleast one of a moveable head; moveable eyes; a moveable torso; one ormore moveable appendages; or any combination thereof.

In Example 90, the vehicle embodied agent system of Example 89 isdisclosed, wherein executing a routine comprises moving a moveableportion.

In Example 91, the vehicle embodied agent system of Example 89 or 90 isdisclosed, wherein executing a routine comprises moving a moveableportion such that the embodied agent's eyes are positioned to face theoccupant.

In Example 92, the vehicle embodied agent system of any one of Examples85 to 91 is disclosed, further comprising one or more sensors,configured to receive information of a vicinity of the vehicle; andwherein executing the routine comprises selecting a point in thevicinity of the vehicle corresponding to a visual cue of the vehicleoccupant.

In Example 93, the vehicle embodied agent system of Example 92 isdisclosed, wherein the visual cue is a gaze direction of the occupant.

In Example 94, the vehicle embodied agent system of Example 92 or 93 isdisclosed, wherein the visual cue is a direction of an upper extremityof an occupant, and wherein the one or more processors are furtherconfigured to detect from the one or more images an upper extremity ofan occupant; and determine the visual cue according to a direction ofthe upper extremity.

In Example 95, the virtual embodiment display system of Example 94 isdisclosed, further comprising selecting a point in the vicinity of thevehicle corresponding to both the gaze direction and the direction ofthe upper extremity.

In Example 96, the vehicle embodied agent system of any one of Examples89 to 95 is disclosed, wherein the one or more processors are furtherconfigured to control the moveable portion to move based on thedetermined gaze direction.

In Example 97, the vehicle embodied agent system of any one of Examples89 to 95 is disclosed, wherein the one or more processors are furtherconfigured to control the moveable portion to move based on the upperextremity position.

In Example 98, the vehicle embodied agent system of any one of Examples89 to 97 is disclosed, wherein the one or more processors are furtherconfigured to control the moveable portion to move based on both thedetermined gaze direction and the determined upper extremity position.

In Example 99, the vehicle embodied agent system of any one of Examples89 to 98 is disclosed, wherein the one or more processors are furtherconfigured to control the moveable portion to move based on a movementof the occupant.

In Example 100, the vehicle embodied agent system of any one of Examples89 to 99 is disclosed, wherein the physical embodiment of theintelligent agent is a robot.

In Example 101, the vehicle embodied agent system of any one of Examples89 to 100 is disclosed, wherein the physical embodiment of theintelligent agent comprises a mechanical likeness of a human being.

In Example 102, the embodied agent system of any one of Examples 89 to100 is disclosed, wherein executing a routine comprises moving amoveable portion such that the embodied agent's eyes are positioned toface a location on which an occupant's gaze is also directed.

In Example 103, the vehicle tag system of any one of Examples 1 through17 is disclosed, wherein vehicle comprises a plurality of vehicleoccupants.

In Example 104, the vehicle tag system of Example 103 is disclosed,wherein the one or more processors are further configured to identifythe plurality of vehicle occupants in the one or more images; select oneof the vehicle occupants according to a selection instruction; determinea gaze direction of the selected vehicle occupant from the one or moreimages; select a display location corresponding to the determined gazedirection; and control the image display device to display a tag at thedisplay location.

In Example 105, the vehicle tag system of Example 103 is disclosed,wherein the one or more processors are further configured to identifythe plurality of vehicle occupants in the one or more images; determinea gaze direction of each of plurality of vehicle occupants from the oneor more mages;

select a display location corresponding to the determined gazedirections; and control the image display device to display a tag at thedisplay location.

In Example 106, the vehicle tag system of Example 105 is disclosed,wherein the display location is selected based on at least one of gazedirections of the plurality of occupants, an angle of a surface of thedisplay location relative to at least one of the plurality of occupants,a relative distortion of the tag for each of the plurality of occupants,or any combination thereof.

In Example 107, the vehicle tag system of any one of Examples 103 to 106is disclosed, further comprising one or more sensors, configured toreceive information of a vicinity of the vehicle; and wherein the one ormore processors are further configured to select a point in the vicinityof the vehicle corresponding to a visual cue of the vehicle occupants.

In Example 108, the vehicle tag system of Example 107 is disclosed,wherein the visual cue is a gaze direction of the occupants.

In Example 109, the vehicle tag system of Example 107 or 108 isdisclosed, wherein the visual cue is a direction of an upper extremityof the occupants, and wherein the one or more processors are furtherconfigured to detect from the one or more images an upper extremity ofan occupant; and determine the visual cue according to a direction ofthe upper extremity.

In Example 110, the vehicle tag system of Example 109 is disclosed,further comprising selecting a point in the vicinity of the vehiclecorresponding to both the gaze direction and the direction of the upperextremity.

While the disclosure has been particularly shown and described withreference to specific aspects, it should be understood by those skilledin the art that various changes in form and detail may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims. The scope of the disclosure is thus indicated bythe appended claims and all changes, which come within the meaning andrange of equivalency of the claims, are therefore intended to beembraced.

What is claimed is:
 1. A vehicle tag system comprising: one or moreprocessors, configured to determine a gaze direction of a vehicleoccupant from image sensor data representing one or more images of thevehicle occupant; select a display location corresponding to thedetermined gaze direction; and control an image display device todisplay a tag at the display location; the image display device,configured to display the tag at the selected display location accordingto the one or more processors.
 2. The vehicle tag system of claim 1,wherein the one or more processors are configured to determine the gazedirection of the vehicle occupant from the one or more images bydetermining from the one or more images a head position and an eyeposition of the vehicle occupant.
 3. The vehicle tag system of claim 1,wherein the one or more processors are configured to determine from theone or more images a head yaw and an eye yaw, and wherein determininggaze direction comprises combining the determined head yaw and eye yaw.4. The vehicle tag system of claim 1, wherein the image display deviceis a projector, wherein controlling the image display device to displaythe tag comprises controlling the projector to project the tag onto theselected display location.
 5. The vehicle tag system of claim 1, whereinthe image display device is a display, and wherein controlling thedisplay to display the tag at the selected display location comprisesdetermining a location on the display that corresponds to the determinedgaze direction and controlling the display to display the tag at thedetermined location.
 6. The vehicle tag system of claim 1, wherein thetag is an embodied agent and/or a physical representation of anartificial intelligence.
 7. The vehicle tag system of claim 1, furthercomprising one or more sensors, configured to receive information of avicinity of the vehicle; and wherein the one or more processors arefurther configured to select a point in the vicinity of the vehiclecorresponding to a visual cue of the vehicle occupant.
 8. The vehicletag system of claim 7, wherein the visual cue is a gaze direction of theoccupant or wherein the visual cue is a direction of an upper extremityof occupant, and wherein the one or more processors are furtherconfigured to: detect from the one or more images an upper extremity ofan occupant; and determine the visual cue according to a direction ofthe upper extremity.
 9. The vehicle tag system of claim 1, wherein thevehicle comprises a plurality of occupants, and wherein the one or moreprocessors are further configured to: identify the plurality of vehicleoccupants in the one or more images; determine a gaze direction of eachof plurality of vehicle occupants from the one or more images; select adisplay location corresponding to the determined gaze directions; andcontrol the image display device to display a tag at the displaylocation.
 10. The vehicle tag system of claim 9, wherein the displaylocation is selected based on at least one of gaze directions of theplurality of occupants, an angle of a surface of the display locationrelative to at least one of the plurality of occupants, a relativedistortion of the tag for each of the plurality of occupants, or anycombination thereof.
 11. A vehicle embodied agent system comprising: aphysical embodied agent; one or more image sensors, configured toreceive one or more images of vehicle occupant; and one or moreprocessors, configured to: determine a gaze direction of the vehicleoccupant from the one or more images; and control the physical embodiedagent to execute a routine according to the determined gaze direction.12. The vehicle embodied agent system of claim 11, wherein the one ormore processors are configured to determine the gaze direction of thevehicle occupant from the one or more images by determining from the oneor more images a head position and an eye position of the vehicleoccupant.
 13. The vehicle embodied agent system of claim 11, wherein theone or more processors are configured to determine from the one or moreimages a head yaw and an eye yaw, and wherein determining gaze directioncomprises combining the determined head yaw and eye yaw.
 14. The vehicleembodied agent system of claim 11, wherein the one or more processorsare configured to determine the gaze direction of the vehicle occupantby determining from the one or more images a head position and an eyeposition of the vehicle occupant.
 15. The vehicle embodied agent systemof claim 11, wherein the physical embodied agent is configured with amoveable portion comprising at least one of a moveable head; moveableeyes; a moveable torso; one or more moveable appendages; or anycombination thereof.
 16. The vehicle embodied agent system of claim 15,wherein executing a routine comprises moving a moveable portion.
 17. Thevehicle embodied agent system of claim 15, wherein executing a routinecomprises moving a moveable portion such that the embodied agent's eyesare positioned to face the occupant.
 18. A method of displaying a tagcomprising: receiving one or more images of vehicle occupant;determining a gaze direction of the vehicle occupant from the one ormore images; selecting a display location or locations corresponding tothe determined or potential gaze direction; and controlling an imagedisplay device to display a tag at the display location.
 19. The methodof displaying a tag of claim 18, wherein determining the gaze directionof the vehicle occupant from the one or more images comprisesdetermining from the one or more images a head position and an eyeposition of the vehicle occupant.
 20. The vehicle tag system of claim 1,wherein the tag is an embodied agent, an avatar, or a physicalrepresentation of an artificial intelligence.